InfiniBand (IB) technology is an advanced, Ethernet-based, packet oriented technology which provides enhanced scalability over traditional Ethernet. The scalability is a very useful feature for high performance computing (HPC) applications where the upgradeability in the number Central Processing Unit (CPU) cores and the type/capability of the CPU core is very important. Due to this need to constantly upgrade and increase the number of CPU cores or “nodes” in a given HPC cluster, InfiniBand is the Input/Output (IO) data interface of choice for such applications. There are many applications where geographically dispersed HPC sites would like to share, in real-time, and in near real-time volumes of data and intermediate results (data). Hence the need to interface these sites via a dedicated and/or shared WAN (wide-area network).
Present State-of-the-Art IB wide area network (WAN) technology is centered around 10 Gbps technology particularly using the 10 GbE local area network (LAN) protocol. This is not efficient from a fiber bandwidth and spectral efficiency point-of-view, and cannot be easily scaled to transporting rapidly increasing volumes of data. At the same time, there is a fortuitous convergence of data communication (i.e. Ethernet) and telecommunication (i.e. SONET/SDH) data rates at 10 Gbps. The continually growing network bandwidth requirements are forcing the industry to look at the next logical step in increased rate of data transport. There is some consensus that it is highly desirable to preserve the convergence of data and telecommunication rates, to leverage component, subsystem and system supplier base commonality, etc. The standards are being pursued by the IEEE Task Force 802.3ba, targeting an Ethernet based solution with a 100 Gbps media access rate.
The present shortcoming can be grouped into three categories. First, there is a general requirement for providing an architecture that can be flexibly adapted to both InfiniBand transport requirements, as well as to whatever next generation higher-speed transport is defined by the Standards groups (such as 100 Gb Ethernet, for example). Satisfying this requirement is advantageous since it leverages a single development effort, higher manufacturing volumes, and the like over a wider application space. It also allows customers to dynamically adapt the same product to fit different needs.
Second, insufficient bandwidth is a major shortcoming of the present designs related to WAN transport of both Infiniband and Ethernet data/telecom traffic. The overall bit rate is really limited to 10 Gbps as of today, generally using 10 GbE/OTU2 based technologies. At the same time, HPC clusters can produce huge amounts of data that needs to be shared. These can include, for example, data sets used for environmental modeling, data sets from hyper-spectral imaging, data sets from high-energy particle colliders, medical genomic research, and the like. Such data sets can reach Petabyte sizes and can use widely deployed specialized distributed file sharing protocols (e.g. Lustre, www.lustre.org). Sharing a Petabyte size file would require approximately 106 seconds or 11.5 days if done with current 10 Gbps data technology. Thus, there is a continual desire for increased transport rate, with current activities focused on 100 Gbps.
Third, there is a general limitation of Infiniband, which is intended only as a short reach (<150 m optical or <5 m electrical) interface between switches connecting computer centers. There is a direct need to provide a seamless and transparent WAN connection for geographically separated computation and storage centers, with required distances ranging from hundreds of kilometers to thousands of kilometers or more.